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Abstract: A machine-vision-based method of locating crops is described in this research. This method was used to provide 
real-time positional information of crop plants for a mechanical intra-row weeding robot. Within the normalized red, green, 
and blue chromatic coordinates (rgb), a modified excess green feature (g-r>T & g-b>T) was used to segment plant material 
from back ground in color images. The threshold T was automatically selected by the maximum variance (OTSU) algorithm 
to cope with variable natural light. Taking into account the geometry of the camera arrangement and the crop row spacing, the 
target regions covering the crop rows were defined based on a pinhole camera model. According to the statistical variation in 
the pixel histogram in each target region, locations of the crop plants were initially estimated. To obtain the accurate locations 
of crops, median filtering was conducted locally in the bounding boxes of the crops close to the bottom of the images. For the 
lateral guidance of the robot, a novel method of calculating lateral offset was proposed based on a simplified match between a 
template and the detected crops. Field experiments were conducted under three different illumination conditions. The results 
showed that the accurate identification rates on lettuce, cauliflower and maize were all above 95%. The positional error as 
within +15 mm, and the average processing time for a 640x480 image was 31 ms. The method was adequate to meet the 
technical requirement of the weeding robot, and laid a foundation for robotic weeding in commercial production system. 
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1 Introduction 


Every year, weed infestation causes huge losses in 
agricultural production over the world, although large 
amounts of labors and herbicides have been used. 
Furthermore, applying of herbicide may threaten 
environmental, food and operator safety, while the cost of 
labor increases year by year. In big farms which 
produce organic crops, it is quite difficult to employ 
enough workers for hand weeding in limited time. 
Non-chemical automatic weeding devices are badly in 


need. 


Inter-row weeds can be treated with traditional 
machineries, while intra-row weeds still remain a 
problem. Robotic mechanical weeding is a potential 
means to reduce the environmental loading of 
agrochemicals in conventional agriculture and replace 
hand weeding in organic agriculture. Thus a research to 
intra-row weeding robot was conducted by our group in 
recent years. Acquiring the location information of the 
crop plants speedily and accurately is the premise of 
performing efficient automated intra-row weeding with 
low crop damage level. Machine vision was chosen as 


the technical means of detecting crop plants and acquiring 


their positional information. The reason was that it was 
more accurate or relatively cost effective compared to 
other approaches, e.g. RTK (real-time kinetics) GPS. 
commercial 


Considering of the requirements of 


production system, the machine-vision-based crop 
location system was expected to be adequate for a 
traveling speed over 2 km/h. This required the 
algorithm to be as simple and fast as possible. 

Many efforts have been made to achieve fast and 
accurate plant location. Tillett et al.!'! used Kalman 
filter to track crop rows as well as to predict plant 
position, and a Mexican hat wavelet to filter the image 
and refine the plant position. The algorithm performed 
well under normal commercial weed infestation levels. 
Zhang et al.!! developed an algorithm based on the 
histogram analysis of plant-pixels and tested on a robotic 
platform. The average time consuming was 20 ms 
which was quite efficient. Hu et al.!°! fitted a sinusoid to 
the lateral pixel histogram to locate crop plants and 
achieved high recognition rate. Weyrich et al.“ 
developed an machine vision system that was able to 
detect the overlapping edges of leaves on NIR image so 
that the leaves were separated and assigned to the 
assessed stem position of plants. Researchers also 
explored the application of artificial intelligence on 
discriminating crop and weed plants by designing 
different kinds Most of the 


classification features were calculated from the color, 


of classifiers, 


texture and shape of plants. Besides using 2D vision 
system, Jin et al.''4! proposed that real-time stereo vision 
system may have better performance when sensing crop 
plants in outdoor lighting conditions and prominent weed 
infestation. 

The aim of this research is to provide a practically 
useful approach to crop sensing for an intra-row 
mechanical weeding robot. Unlike many previously 
proposed methods?:'>!, we did not extract the crop rows 
before locating individual plants, while we use 
perspective relation to determine the crop row area in the 
image, which could reduce the searching scope when 
processing image. Moreover, it made the algorithm 
more robust to the obstacle due to inter-row weeds. 


When obtaining accurate crop position, median filtering 


was conducted within the bounding boxes of crops, in 
order to reduce the disturbance of the weeds near the crop 
plants. Calculation of lateral offset did not depend on 
fitting of crop row center lines. Rather, it was 
determined by the best match between the centroids of 
the crops near the image bottom and a template. To 
demonstrate the capabilities of the method, experimental 
work was carried out on different crops under natural 


light. 
2 Materials and method 


2.1 Materials 

The machine vision system mainly consisted of a 
752x480 color camera (Do3think CM036) with a 4 mm 
lens (AZURE-0420mm), an industrial control computer 
(NORCO RPC-208) with a 2.8 GHz Intel Core 2 Duo 
E7400 processor, and an 8-inch touch display. The 
resolution of the images was 640x480 after setting the 
interest area of camera. The image processing software 
was developed in Microsoft Visual Studio 2010, based on 
the camera’s SDK. 

The system was mounted on a robotic intra-row 
weeding experimental platform (Figure 1) which was 
connected to a tractor through a 3-point linkage. The 
camera was mounted 1.85 m above the soil surface, and 
was tilted forward at an angle of 20.8° from the vertical 
orientation. This arrangement allowed three crop rows 
to be viewed and at least three crops in each row to 
appear in the image. Cauliflower, lettuce and maize, 
were chosen as sample crops for analyzing and field trials. 
were all 


The sample crops transplanted with a 


semiautomatic transplanter. The nominal row spacing 
was set to 0.5 m, while the nominal plant spacing varying 
from 0.4 m to 0.5 m. 

The intra-row weeding end-effectors of the platform 
were three C type cultivation blades modified from the 
rotating disc described by O’Dogherty et al.° and 
Huang et al.!7, These blades moved along the crop row 
under the ground surface to remove the in-row weeds by 
cutting the roots or stems when the machine went ahead. 
Each of blades could rotate about a substantially vertical 
axis driven by a 48v/300w electric motor. To prevent 


crop damage, the blades rotated when they got close to 
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crops so that the cut-out sections would face the crops 
and make room for them (Figure 2). Thus the machine 
vision system had to provide the real-time distances 
between the approaching crops and the blades. In order 
to keep the blades working within the intra-row areas, the 
machine vision system also had to measure the lateral 


offset between the crop rows and the blades. 


Camera Touch display 


Computer 


Weeding blade Control box 


Figure 1 The robotic intra-row weeding experimental platform 


*:—Dircetion of-tenyct 


Figure 2. Weeding blade rotates to avoid crop plant 


2.2 Method 
2.2.1 


The overall procedure of image processing algorithm 


Color to monochrome image 
is depicted in Figure 3. The first step after reading an 
image was to discriminate plant material from the 
background based on their difference in color feature. A 
variety of strategies for this step were enumerated by 
Montalvol!®!, One of the most widely used green plant 
extraction indexes was the excess green index (ExG) 
2g-r-b, which was introduced by Woebbecke et al.!!9!. 
This index was less sensitive to the intensity of the 
illuminating source and its angle with the target surface, 


but was sensitive to noise at pixels with low intensity. 


Image acquisition 


Color to gray image 
(min(G-R, G-B) 


Image thresholding (OTSU) 


Crop row region estimation 


Refining the pixel histogram 
By scaling, subtraction and 
smoothing 


Initial location of crop plants 


Local median filtering Estimation of lateral 
offset depending on a 
match with the nominal 


row structure 


Distance between the 
end effectors and the 
approaching crops 


Calculating of plant centroids 


Figure 3 Crop location method architecture 


According to the analysis of red, green and blue 
channels from color images of different sample crops and 
the background (Figures 4c and 4d), we chose a modified 
ExG index defined by Equation (1) to transform the color 
images into gray images. 


may nee (GƏR & G>B) 


1 
0 (1) 


where, R, G and B are the intensities of the red, green and 
blue channels of an image pixel; r, g and b are the 
normalized chromatic coordinates of R, G and B; M is the 
gray value of the pixel in the resulting image. 

The gray images were transformed to monochrome 
images (Figures 4e and 4f) by applying the threshold 
This 


algorithm is self-adjustable, dealing well with images 


automatically selected by OTSU!” algorithm. 


captured under different illumination conditions such as 
sunny or cloudy days’, 
2.2.2 Estimation of target regions 

As the crops grew in rows, searching of individual 
What’s 


more, the inter-row weeds may act as a kind of noise 


crop plants could be limited in the in-row areas. 


disturbing the correct positioning of crop plants. In 
some cases, weeds grew neatly in rows between the crop 
rows and became a prominent feature. Figure 5 shows a 
luxuriant growth of weeds in the depressions on the 
ground caused by the depth wheels of the transplanter. 

In order to prevent the inter-row weed disturbance 
and reduce the time consuming of algorithm, estimation 


of the in-row areas were done before doing crop location. 
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Target regions covering the in-row areas were defined 
according to the geometry of the camera arrangement, the 
crop row spacing, size of the crops and the calculated 
lateral offset of last image. 

The model of the camera over the crop plants is 
shown in Figure 6. It was assumed that row direction 
does not deviate too much from the y-axis in the image 
(normally no more than +4°). Hague et al.!?! used the 
perspective relationship between world coordinates and 
image coordinates to determine the boundaries of the 


target regions. 


a. Sample image of cauliflower b. Sample image of maize 
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c. Means and standard deviations of RGB channels of cauliflower and soil in (a) 


300 


E Standard deviation W Mean value 


250 


200 


Intensity 
a 
© 


Red Green Blue 
Color channel 


d. Means and standard deviations of RGB channels of maize and soil in (b) 
’ ~~ i P4 
s “ 
eS.) 
4 ” & ad 
mal %6 
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Figure 4 Sample images and segmentation results 
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a. Side elevation of the camera model 


Imaged area 


b. Field of view 


Figure 6 Model of the camera over crop plants 


As the vertical ordinate y; decreased down the image 
in this vision system, it was opposite to that described by 
Hague et al.?! Therefore, the relationship between the 
abscissa value x; in the image coordinate system (Figure 
6a) and its value x, in the world coordinate system 
(Figure 6b) was described as Equation (2). Since the 
angular deviation of crop rows was ignored, points on a 
boundary line of a crop row area should have a same 
distance to the vertical axis in the world coordinate 
system. Giving a fixed value of x, according to the row 
spacing and row width, equation of the boundary line in 
the image coordinate system could be obtained from 
Equation (2). The blue lines in Figure 7 show the target 


region estimation result in an in-door simulative scene. 
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I 
x,Lf. —(y, -> tang] 
x= 2 ph (2) 
Z 2 


where, fe is the focal length of the camera lens which was 
obtained by a calibration procedure based on the 
calibration toolbox of MATLAB; Z and J, are the width 


and height of the image in pixels respectively. 


Figure 7 Estimation of target regions in a simulative scene 


a. Monochrome image of maize with much noise 
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c. Histogram after smoothing 


2.2.3 Localization of individual plants 

Once the target regions were defined, the black pixels 
of every horizontal line within each target region of the 
monochrome image were counted. Thus a pixel 
histogram was formed. For better demonstration, an 
image with much noise was chosen as an example (Figure 
8a) and Figure 8b is the related pixel histogram. 

Because of perspective, further plants appear smaller 
than the ones close to the camera even though they are 
similar in size. In order to eliminate or reduce the 
influence of perspective, number of black pixels in each 
line was multiplied by a factor: 

N,=N,x hy (3) 

W, 
where, N; is the number of black pixels in a horizontal 
line within the target region and i is the vertical ordinate 
of the line; W; is the width of the target region at line i (i 


increases from 0 to 479 up the image). 


480 
450 
420 
390 
360 
330 
300 
270 
240 
210 
180 


Line number 


30 60 90 12030 60 90 12030 60 90 120 
Quantity of black pixels 


b. Pixel histogram of (a) 


d. Delimited crop areas 


Figure 8 Histogram of maize image with much noise crop areas delimitation 


To erase the non-crop black pixels, the average of the 
black pixel numbers in each region was subtracted from 


the black pixel number of each line: 


479 479 
sy, 
i i N, > = (4) 
480 480 
After that, the pixel histogram was smoothed with a 
one-dimensional mean filter defined as Equation (5). 


Figure 8c shows the smoothed pixel histogram. 
1 i+4 
N,==x DN, (5) 
9 TA 

The black areas left in the smoothed pixel histogram 
indicate the vertical location of the crop plants, though 
there still are some non-crop areas. By applying an area 
threshold to the black areas, the non-crop areas were 
eliminated. Thus the upper and lower boundaries of the 
individual crops can be obtained. With the boundaries 
of target regions, areas containing the crops were 
delimited (Figure 8d). 

In order to erase the single and isolated pixels in the 
delimited crop areas which may influence precise 
positioning, median filtering was conducted locally 
within the crop areas. Since the weeding robot only 
required the distance to the nearest crop in each row, 
there was no need to acquire the accurate positions of all 
the crops. Therefore, only the lowest crop area in each 
crop row in the image was filtered. This significantly 
reduced the time consuming of the algorithm and 
redundancy of information. By calculating the centroids 
of the black pixels in the filtered areas, accurate locations 
of crop plants were obtained (marked with blue crosses 
shown as Figure 8d). The coordinates of the centroids 
in the world coordinate system defined in Figure 6b were 
obtained as Equations (6) and (7). Then the distances 
between the weeding blades and the approaching plants 
were calculated. 

Z(x; — Ey 
x, = r (6) 
f-0: -5ng 


Z( By 

oe 

Yu = 2 (7) 
f.cosd—(y, = ene 


2.2.4 Estimation of lateral offset 


Many efforts have been devoted to machine 
vision-based automatic guidance of robots, machines and 
autonomous vehicles within agricultural environments by 


detecting crop rows?3?71, 


Hough transform and the least 
squares method were very commonly used for crop line 
detection. However, these two methods are relatively 
computationally intense. According to the work 
characteristics of the weeding robot, we devised an 
efficient approach to obtain the lateral offset of the 
end-effectors. This approach did not depend on the 
location of the crop rows, but the location of the plants 
nearest to the end-effectors. 

With the relationship expressed by Equation (6), the 
abscissas of the centroids in the image coordinates were 
converted in to abscissas in the world coordinates. A 
template symbolizing the nominal row structure (the blue 
lines in Figure 9a) was used to find the expected position 
of the intra-row weeding blades. This was done by 
matching the template with the calculated crop positions. 


Row 1 Row 2 Row 3 


a. Illustration of template 


Good Acceptable Inacceptable 


b. Assessment of different blade positions 
Figure 9 Illustration of template and assessment of blade 


positions 


When the robot was working in the field, centers of 
the blades were expected to be right in the centers of the 
crop rows. If a blade was placed on the left of the crop 
row center, coverage of the blade in the in-row area 
would drop. On the other hand, if the blade was placed 
on the right side of the row center, it may cause damage 


to the crops, which was inacceptable (Figure 9b). Based 
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on this principle, the best match would achieve only when 
at least one of the blue lines in Figure 9a coincided with 
the related row center while none of the others lay on the 
right of the row center (the practical row spacing may 
vary slightly along the rows, the three blue lines can 
hardly match the rows accurately at the same time). 
Figure 9a illustrates an example of best match. 
Therefore, we only consider three situations that each of 
the blue lines overlapped with the corresponding crop 
Each of the 


evaluated according to the relative positions of the 


center respectively. situations was 


non-overlapped lines and row centers, as described by 
Equations (8), (9) and (10). 


l (4% 2x +D, x 2x, +2D) 

m =d 2 1 pi 1 (8) 
0 
l, (%2x,-D,x,=x,+D) 

m=] 1 2 3 2 (9) 
0 

M, “1 (x, 2x -—2D, x, => x, —D) (10) 


where, Mı, M2, M3 are the evaluation of the match under 
the three assumed situations; crop rows are numbered 1, 2 
and 3 from left to right in the image; x, x2 and x3 are the 
calculated abscissas of the crop centroids; D is the 
nominal row spacing; the value 1 indicates the 
achievement of best match. 

After the best match was obtained, the lateral offset 
between the expected and practical positions of the blades 


was acquired subsequently by calculating the abscissa of 


the template center in the world coordinates (The 
practical center of the weeding blade set was at the origin 


of x- axis both in the image and world coordinates). 
3 Results and discussion 


To inspect the location accuracy of the system, an 
in-door test was done by comparing data obtained in 
different ways. A series of distances from the crops to 
the weeding blades and lateral offsets were measured 
manually and by the machine vision system. In order to 
facilitate pointing out the centroids manually, three $85 
mm green discs (Figure 10) was used to instead of crop 
plants. The discs were fixed on an aluminum bar, 
separated from each other by equal spaces (500 mm). 
By changing the position of the discs in the view area of 
the camera, we got a set of data (Table 1) which showed 
the location error of the system was within +15 mm. 
The system accuracy was thought to be adequate for 
avoiding crop damage. Sources which affected the 


location accuracy mainly included the deviation in 


physical set up and lens distortions. 


Figure 10 Materials used in the in-door test 


Table 1 Results of in-door test mm 

Number of data 1 2 3 4 5 6 7 8 9 10 
Distance 1 289.0 240.0 492.0 443.5 359.0 331.0 248.0 321.0 317.0 232.0 
Distance 2 292.0 288.0 495.0 442.0 354.0 337.5 247.0 325.0 318.0 227.5 

Measured by man 

Distance 3 294.5 337.5 498.0 438.0 350.0 341.0 245.5 329.0 320.0 223.5 

Offset 46.0 —65.0 57.5 36.0 55.0 70.5 92:5 —86.0 60.0 82.0 
Distance 1 300.0 252.0 490.0 446.5 364.0 343.0 260.0 332.0 326.0 244.0 
Distance 2 302.0 296.0 496.0 443.0 360.5 346.5 258.0 334.0 326.0 242.0 

Measured by system 

Distance 3 308.0 344.0 500.0 441.0 360.0 353.5 260.0 340.0 332.0 238.0 

Offset 35.5 =75.5 51.0 35:5 43.5 =a —61.0 —93.0 61.5 T33 
Distance 1 —11.0 —12.0 2.0 3.0 5.0 12.0 12.0 11.0 —9.0 -12.0 
Distance 2 —10.0 8.0 1.0 1.0 6.5 9.0 11.0 9.0 8.0 14.5 

Location error 

Distance 3 =13.5 6.5 2.0 3.0 10.0 12.5 14.5 11.0 12.0 14.5 

Offset 10.5 10.5 6.5 0.5 11.5 7.0 8.5 7.0 -1.5 8.5 


The performance of the system was also evaluated in 


field environment at Tongzhou, Beijing. The field 


experiments were conducted at approximately 14:00 and 


16:30 under different directions of sun light on 30th April, 
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2015, and 14:30 on 6th May when it was cloudy. The 
tractor was driven at approximately 1.5 km/h. Before the 
experiments, the sample crops, cauliflower, maize and 


lettuce, had grown for 13 d since they were transplanted. 


Numbers of the crops were 225, 287 and 257 respectively. 


Weeds in the field mainly consisted of amaranthus 
lividus, purslane and digitaria sanguinalis. The average 
weed density in the field was 126 per m? on 30th April 
and 266 per m?on 6th May. The first weeding treatment 
is usually conducted at this stage in normal vegetable 
production. The weeding blades of the experimental 
platform were taken off to prevent potential crop damage. 
The detected crops were marked with blue crosses in the 
image shown on the touch display. 

Table 2 shows the results of field experiments. It is 
shown that the correct recognition rates were all above 
95%. Generally, the system achieved better performance 
on cloudy day than in direct sun light. However, change 


in direction of the sun light did not affected significantly 


on the correct recognition rate. Lettuce was better 
detected than the other two crops since it had broad 
leaves and strong contrast with soil in color. 
Cauliflower was not so well segmented from the back 
ground as lettuce. Maize was more likely to be 
unrecognized since it has long thin leaves and may be 
divided into separate parts (Figure 11a). Besides, some 
of the maize had leaf disease that did not appear green 
11b). A of the 


misrecognition was due to large weeds that occasionally 


(Figure significant component 


emerged near or between the crops. Those weeds were 
often recognized as crops (Figure 11c). However, 
inter-row weeds could hardly influence the system. An 
important factor of failing to recognize crop plants was 
using of a fixed area threshold, which may discriminate a 
few undersized plants as weeds. Direct sun light caused 
more noise and less complete extraction of crops in the 


background segmentation procedure. 


Table 2 Results of the field experiment 


Non crops recognized 


Total numbers of Total numbers of Correct recognition 


Time Crop as crops inne poured cneps incorrect recognitions crops rate/% 

Cauliflower 3 7 10 225 95.6 

2:00 pm, ; 
30th April Maize 2 11 13 287 95:5 
Lettuce 2 257 98.4 
Cauliflower 3 225 95.1 

4:30 pm, F 
30th April Maize 2 287 96.2 
Lettuce 2 257 98.4 
Cauliflower 4 225 96.4 

2:30 pm, z 
6th May Maize 3 287 96.9 
Lettuce 4 257 98.1 

t y 


a. Maize divided into two parts 


Figure 11 


The software automatically recorded the time soon 
after it finished processing a frame of image. The 
average time consuming of the algorithm was about 


31 ms. Therefore, the machine vision system is 


b. Unhealthy maize 


c. Big weed close to crop 


Some special cases that affecting recognition rates 


adequate to provide real-time crop location when the 
robot is working at a forward speed of 1.5 km/h or higher, 
e.g. 2 km/h. In fact, the system was able to achieve a 
good performance with a speed over 2.5 km/h, if the 


platform traveled on a flat ground. However, when the 
platform was working on a complex field surface, with 
the increase of speed, intensified mechanical vibration 
caused greater location error. In that case, it was 
difficult for the platform to maintain a low crop damage 
rate. Thus, the authors considered the maximum speed 
for operation in field to be 2 km/h. As a result of 
calculation, the operation efficiency would be 2.4 hm?/d 
(3x0.5 mx2 km/hx8 h=2.4 hm?) if it works 8 hours per 
day under the speed of 2 km/h (Note that 3 is number of 
blades, 0.5 indicates crop row spacing). That is about 
34.3 times of the efficiency of a labor (usually 0.07 


hm?/d). 
4 Conclusions 


A novel method of locating crops is described and 
evaluated. The main contribution is to provide a fast 
and accurate approach to obtain the real-time positional 
The 


algorithm was designed to cope with variable out-door 


information of crops close to the weeding robot. 


illumination and weed disturbance in normal production 
system. To reduce the computational burden, very few 
computationally intensive algorithms were used. 
Nevertheless, the method was robust to variation in 
illumination and crop species by taking advantage of 


color and position features of crops. Location error of 


the crop detection system was maintained within +15 mm. 


Average time consuming for processing a 640x480 image 
was 31 ms, which was quite efficient compared with most 
of previously proposed methods. Results of field 
experiments indicated the method was effective to detect 
over 95% of cauliflower, maize and lettuce under natural 
light. As a conclusion, the method is adequate to meet 
the technical requirement of intra-row weeding robots, 


and hopeful to be used in commercial production system. 
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